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Se \\ Testing Overview 


Prof. Philip Koopman 


"In September of 1962 ... an $18 million rocket had been destroyed 
in early flight because “a single hyphen was left out of an instruction 
tape."... The nature of programming being what it is, there is no 
relationship between the ‘size’ of the error and the problem it 
causes. Thus, it is difficult to formulate any objective for program 
testing, short of ‘the elimination of all errors" - an impossible job. " 


- Gerald M. Weinberg goo.gl/w42tqZ © 2020 Philip Koopman 1 
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= Anti-Patterns: 
e Primarily ad hoc, informal testing 
e No defined notion of testing coverage 
e No defined pass/fail test results 
















EXPECTED — ACTUAL 
RESULT BEHAVIOR 
= Testing executes software 
e A workload is applied to SW under test .< 7 
FAIL PASS 


e An oracle generates expected results 
e The test passes if SW matches oracle 
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= “Smoke Test” 
e Quick check to see if the software runs at all 
e Valuable pre-test if it exercises major functions 


= Exploratory Testing 
e Expert tester beats on the system 
e Can find low hanging fruit to fix if tester is good 





= Problems with unstructured approaches 
e Hard to know how much youve tested 
e Test coverage is often subjective 
e Effectiveness heavily dependent upon tester skill 


© 2020 Philip Koopman 4 


Carnegie 


Methodical Testing Approaches . ES 


=m Black Box Testing 
e Use requirements to guide test design 
e Exercise all code functions 
m White Box Testing 
e Use structure of code to guide test design 
e Exercise all paths through code 
= Need both black box and white box testing 
e Black box testing can miss implementation corner cases 
e White box testing doesn't exercise missing functionality 
m Coverage: how completely have you tested? 
e White box coverage: fraction of code exercised in test 
e Black box coverage: fraction of requirements exercised 
e Other coverage: e.g., fraction of state chart arcs traversed 
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Testing Across Development Cycle Pa 


= Testing within V process Testers Of ‘Dangerously Defective’ Autopilot 
e Unit test: single executable procedure ; 

Integration test: do modules play nice? 

Software test: exercise system or subsystem 


Acceptance test: whole-product test with 
customer use scenarios 


=m Other types of testing 

Beta test: see if representative users discover defects 
Regression test: test to see if a previously fixed bug comes back 
Performance testing: measure capacity and identify bottlenecks 
Robustness testing: what happens with invalid inputs? 

Security testing: can an attacker penetrate the system? 

Fault injection: what happens when a component fails? 





Black Tesla https://goo.gl/5UU75u 
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Testing Philosophy Lita 


= Testing finds bugs 
=> but why are you finding bugs? 





= Testing Low Quality Software: 
e Find bug, remove bug, iterate until out of time 
e How many bugs are left? 


= Testing Safety Critical Software: 
e All bugs removed via peer review, unit test, integration test 
— Testing is to prove hypothesis that software is essentially perfect 
e Product-level tests should never find a SW bug 
— If your find a bug, your process is broken 
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= Get high testing coverage 
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Testing Best Practices Mellon 


University 





White box — most or all code paths 
Black box — all requirements 
Exploratory testing to reduce risk 

— Unexpected functionality 

— Low hanging fruit for buggy software 
Other types of testing as appropriate 
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TIMING AND SEQUENCING 


= Testing pitfalls 


Testing all data values is impracticable 
— 2(3*82) = 7.9 * 1028 input values for: proc( int x, int y, int z) 


e Creating an oracle is the hard part of scaling up test generation 
e 100% coverage is not 100% tested 


100% passing tests is not 100% bug-free © 2020 Philip Koopman 8 
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=m Good starting point for more info: 
e http://www.softwareqatest.com/index.html 

















= Useful references: saniee' : SS 

e Kaner, Cem, Jack Falk, and Hung Quoc Nguyen. Testing 
Teaiing eatin porn, semnd Jee a Computer 
nternational Thomson Computer Press, - y 
OR John Wiley & Sons, Inc., 606, ISBN 0-471-35846-0. Software 

e Beizer, Boris. Black Box Testing. cs 6=—h 
John Wiley & Sons, 1995, ISBN 0-471-120904-4. a “na 

e Free Training: http://www.testingeducation.org/BBST/ ta 


Hung Quoc Nguyen 


“f ucnrd ths, ucn gt a gd jb n sftwr tstng.. - Anonymous 


“Failure is not an option. It comes bundled with the software.” - Anonymous 
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Soo oon"! 
To catch them is your real test, to train them is your cause. 


Catching ‘em All 


The Definitive Guide 


O RLY? @ ThePracticalDev 
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WOULD YOU GUESS 


THUMP # VRRRRARER RRR eR ence 
THIS WEIGHS AS MUCH | PLEASE FASTEN 
AS A SMALL ADULT? 


your Sear Bez || T LOVE SELF-DRIVING CARS. ~ 
#CLICK# ,.. WHOSE CAR LAS THAT? | 


DUNNO, BUT THEY SHOULDN7T- 
HAVE LEFT IT RUNNING. 
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https://xkced.com/1559/ 
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